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ABSTRACT 



A method for estimating the motion between a first image 
and a second image in a system for creating a panoramic 
image from a plurality of images taken by a camera, the 
method comprising the steps of: downsampling a first image 
in a first direction and in a second direction; downsampling 
a second image in a first direction and in a second direction; 
filtering the first and the second image so as to filter out any 
global illumination changes between the first image and the 
second image; calculating a first displacement along the first 
direction between the first downsampled image along the 
first direction and the second downsampled image along the 
first direction; and calculating a second displacement along 
the second direction between the first downsampled image 
along the second direction and the second downsampled 
image along the second direction. In an alternate 
embodiment, a device and computer readable medium cor- 
responding to the above method is described. 

18 Claims, 25 Drawing Sheets 
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MOTION ESTIMATION FOR PANORAMIC 
DIGITAL CAMERA 

CROSS-REFERENCE TO RELATED 

APPLICATIONS 5 

This application relates to technology similar to U.S. 
patent applications identified by attorney docket numbers 
99-RB-105 now Ser. No. 09/477,037, 99-RB-106 now Ser. 
No. 09/477,036, 99-RB-107, Ser. No. 09/476,652 now U.S. 
Pat. No. 6,456,323, 99-RB- 108 now Ser. No. 09/477,919, 
and 99-RB-123, Ser. No. 09/477,118 now U.S. Pat. No. 
6,618,511, all being filed concurrently herewith and com- 
monly assigned herewith to STMicroelectronics Inc. and 
which are hereby incorporated by reference in their entirety 
hereinto. 

Partial Waiver of Copyright 

All of the material in this patent application is subject to 
copyright protection under the copyright laws of the United 20 
States and of other countries. As of the first effective filing 
date of the present application, this material is protected as 
unpublished material. 

However, permission to copy this material is hereby 
granted to the extent that the copyright owner has no 25 
objection to the facsimile reproduction by anyone of the 
patent documentation or patent disclosure, as it appears in 
the United States Patent and Trademark Office patent file or 
records, but otherwise reserves all copyright rights whatso- 
ever. 30 

BACKGROUND OF THE INVENTION 
1. Field of the Invention 

The invention disclosed broadly relates to the field of 35 
image processing and more specifically to image processing 
in a digital camera for taking panoramic pictures. 
2. Description of the Related Art 

Today, panoramic photography is accomplished in vari- 
ous ways. One is to use a still camera mounted on a tripod 40 
to take a succession of shots as the camera is pivoted around 
the tripod. In some cameras, a wider than usual strip of film 
is exposed with special movable optics. 

In other cameras, conventional format film, such as 35mm 
film, is masked during the exposure in the camera to provide 45 
a panoramic effect. The effect is panoramic but the whole 
exposure is limited by the field of view through the lens. 

Other techniques for creating panoramic photography 
include to physically cut and paste together strips of exposed 
film by carefully aligning boundaries between edges of film. 50 

The benefits of electronic photography have led to the 
growth of digital cameras, that, unlike their film-based 
counterparts, store images captured in memory into digital 
memory such as flash memory. To provide panoramic pho- 
tography effects, these digital cameras can interface with 55 
personal computers for joining together two or more images 
into one image to provide a panoramic effect by joining edge 
boundaries of Images. One such system is disclosed in U.S. 
Pat. No. 6,682,197, by named inventors Omid A. 
Moughadam, Stuart R. Ring, and John R. Squilla, entitled 60 
"Electronic Panoramic Camera For Use With An External 
Processor". 

Complicated panoramic digital cameras are available that 
rely on position sensors or satellite communications for 
determining position coordinates. These position coordi- 65 
nates are used to help combine the panoramic images. The 
process of combining scenes taken from different camera 
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orientations is known as "Image Stitching." One such sys- 
tem is disclosed in U.S. Pat. No. 5,262,867 by named 
inventor Kiyonobu Kojima entitled "Electronic Camera and 
Device for Panoramic Imaging and Object Searching" 
issued on Nov. 16, 1993. 

A panoramic camera with a memory device for storing 
data from a previously photographed portion of an object 
and a control device for enabling the display device to 
substantially display both the image to be photographed and 
the image already photographed and stored in the memory 
space is described in U.S. Pat. No. 5,138,460 by named 
inventors Egawa and Akira entitled "Apparatus for forming 
Composite Images" issued on Aug. 11, 1992. 

Although these techniques are useful, they are not without 
their shortcomings. One shortcoming is the expense and 
complication arising from the need for orientation and 
position sensors for determining picture orientation. 
Accordingly, a need exists for a device and method to create 
panoramic pictures without the need and expense of position 
sensors. 

Another shortcoming of the current panoramic cameras is 
how difficult it is to properly overlap a region between two 
adjacent frames of a panoramic image. Too much overlap 
results in wasting memory in the digital camera with redun- 
dant information. 

Another shortcoming with current panoramic cameras is 
their inability to guide the user where the correct edge 
overlap is required for creating a panoramic picture align- 
ment from two or more images. Accordingly, a need exists 
to overcome this problem and to guide the user of a camera 
using a method and apparatus to overlap two or more images 
to provide a correct panoramic picture. 

Another shortcoming of the current panoramic cameras is 
the requirement to overlap regions of two or more images. 
It is common with panoramic image generation to stitch 
together two or more images. Problems at the boundaries of 
two images include color distortion, perspective distortion, 
pixel adjacency and skipped pixels at the edges when joining 
two or more images. Accordingly, a need exists for a method 
and apparatus to overcome these shortcomings. 

Another shortcoming of the current panoramic cameras is 
their inability to integrate these mechanisms with other 
products. The bulk, expense, and complication make these 
designs difficult to integrate. Accordingly, a need exists for 
a method and apparatus to allow easy integration of pan- 
oramic capabilities into other electronic devices. 

Another shortcoming is the inability to easily capture two 
or more images to create a panoramic scene without the need 
for expensive computational resources in the camera and 
without the need for expensive position sensors. 
Accordingly, a need exists for a method and apparatus to 
provide a camera which can capture two or more images so 
as to create a panoramic image. 

Another shortcoming is the requirement of having two 
distinct buffers to transform the captured image from one 
coordinate system to another coordinate system, such as 
rectilinear coordinates to cylindrical coordinates. The 
expense and space required for two buffers to coordinate 
transformations is undesirable. Accordingly, a need exists to 
provide a method and apparatus that can transform images 
from one coordinate system to another coordinate system, 
without the use of two buffers. 

Another shortcoming with current panoramic cameras is 
the perspective of the series of images relative to each other 
is lost. One known procedure uses sub-sampling. The pic- 
ture is scaled down so that the horizontal dimension will 
match the horizontal extent of the view-port. However, the 
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picture must become a narrow stripe in order to maintain the 
correct aspect ratio. This will produce a great loss of vertical 
resolution. Another solution is to scroll the panoramic image 
horizontally by a fixed amount. In this case no sub -sampling 
is used, provided that the vertical dimension of the picture 5 
will fit the vertical dimension of the view-port. Accordingly, 
a need exists to provide a method and apparatus to create a 
moving display of still pictures with perspective without the 
need to sub-sample or to horizontal scroll. 

Still another shortcoming with current panoramic cameras 10 
is the requirement that all the image processing electronic 
circuitry for the final stitching together of one or more 
images in a series into a single panoramic image is inte- 
grated into the camera. Many times, the added expense, the 
additional weight and the additional size of the electronic 15 
circuitry makes the camera unwieldy to carry and operate. 
Moreover, the additional electronic circuitry makes the 
camera more expensive to manufacture. Accordingly, a need 
exists for a method and apparatus to enable the preview of 
a series of adjacent images to form a panoramic, and to 20 
enable the final image stitching processing on a remote 
device such as a personal computer. 

SUMMARY OF THE INVENTION 

A method for estimating the motion between a first image 25 
and a second image in a system for creating a panoramic 
image from a plurality of images taken by a camera, the 
method comprising the steps of: downsampling a first image 
in a first direction and in a second direction; downsampling 
a second image in a first direction and in a second direction; 30 
filtering the first and the second image so as to filter out any 
global illumination changes between the first image and the 
second image; calculating a first displacement along the first 
direction between the first downsampled image along the 
first direction and the second downsampled image along the 35 
first direction; and calculating a second displacement along 
the second direction between the first downsampled image 
along the second direction and the second downsampled 
image along the second direction. 

In an alternate embodiment, a device and computer read- 40 
able medium corresponding to the above method is 
described. 

BRIEF DESCRIPTION OF THE DRAWINGS 

The subject matter which is regarded as the invention is 45 
particularly pointed out and distinctly claimed in the claims 
at the conclusion of the specification. The foregoing and 
other objects, features, and advantages of the invention will 
be apparent from the following detailed description taken in 
conjunction with the accompanying drawings. 

FIG. 1 is a block diagram of a digital still camera 
according to one aspect of the present invention. 

FIG. 2 is a block diagram of the picture stitching device 
of FIG. 1 in another embodiment integrated as a standard 55 
cell in a semi-custom semiconductor device, according to 
the present invention. 

FIG. 3 is a block diagram of a series of individual 
horizontal still images comprising a panoramic image with 
overlapping regions, according to the present invention. 60 

FIG. 4 is a flow diagram of the overall image acquisition 
process of the horizontal images of FIG. 3 using the digital 
still camera of FIG. 1, according to the present invention. 

FIG. 5 is an illustration of a Previous Picture for demon- 
strating the composite overlay and perspective correction of 65 
the preview strip and the overall image acquisition process 
of FIG. 4, according to the present invention. 
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FIG. 6 is an illustration of a selected overlap region of 
FIG. 5, according to the present invention. 

FIG. 7 is an illustration of the corrected perspective of 
overlap region of FIG. 6 for aligning the Ideal Next Picture, 
according to the invention. 

FIG. 8 is an illustration of the Current Picture with an 
Ideal Next Picture guided by the overlap generated in FIG. 
7, according to the present invention. 

FIG. 9 is a high level block diagram of a picture stitching 
device of FIG. 1, according to the present invention. 

FIG. 10 is a detailed flow diagram of the stitching 
overview of FIG. 4, according to the present invention. 

FIG. 11 is an illustration of alpha blending mixing 
between the preview strip and the Current View, according 
to the present invention. 

FIG. 12 is an illustration of the results of alpha blending 
of FIG. 11 with a constant ratio, according to the present 
invention. 

FIG. 13 is an illustration of the results of alpha blending 
of FIG. 11 with a progressive ratio according to the present 
invention. 

FIG. 14 is an illustration of the results of interlace 
blending of FIG. 11 using blocks, according to the present 
invention. 

FIG. 15 is an illustration of the results of interlace 
blending of FIG. 11 using lines, according to the present 
invention. 

FIG. 16 is an illustration of the results of alpha blending 
with interlace blending of blocks of FIG. 15, with the 
vertical dimension equal to the vertical dimension of the 
picture according to the present invention. 

FIG. 17 is an illustration of the results of alpha blending 
with interlace blending of only some of the blocks of FIG. 
15, according to the present invention. 

FIG. 18 is an illustration of the areas of measurement for 
two adjacent figures, such as images A and images B of FIG. 
3, according to the present invention. 

FIG. 19 is an illustration of the individual frame buffers 
laid out on a panorama, according to the present invention. 

FIG. 20 is an illustration of the panorama of FIG. 19 saved 
as fixed-sized strips, according to the present invention. 

FIG. 21 is a detailed block diagram of the picture stitching 
device of FIG. 9, according to the present invention. 

FIG. 22 is a block diagram of the Alignment Preview 
architecture of FIG. 21, according to the present invention. 

FIG. 23 is an illustration of the capturing of two rectilin- 
ear image planes and the relationship to a cylindrical circle 
of a fixed focal point f, according to the invention. 

FIG. 24 is an illustration of pixels in the overlap region of 
the current picture of FIG. 23 being projected back to the 
Previous Picture of FIG. 23 in the overlap region, according 
to the present invention. 

FIG. 25 is an illustration of pixels of the selected overlap 
region of source picture of FIG. 7 that cannot be perspec- 
tive ly corrected, according to the invention. 

FIG. 26 is a block diagram of an alternate embodiment of 
FIG. 9 with two optional blocks used to improve the 
perspective correction algorithm performance, according to 
the present invention. 

FIG. 27 is an example of a limit area copy from a Previous 
Buffer to a Current Buffer of FIG. 26 during a first pass, 
according to the present invention. 

FIG. 28 is an example of a limit area copy from a Previous 
Buffer to a Current Buffer of FIG. 26 during a second pass, 
according to the present invention. 
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FIG. 29 is an example of a limit area copy from a Previous 
Buffer to a Current Buffer of FIG. 26 during a third pass, 
according to the present invention. 

FIG. 30 is an example of a limit area copy from a Previous 
Buffer to a Current Buffer of FIG. 26 during a fourth pass, 5 
according to the present invention. 

FIG. 31 is an example of areas of Current Buffer for which 
there is no information, according to the present invention. 

FIG. 32 is an illustration of rectilinear and cylindrical 
images, according to the present invention. 

FIG. 33 is a schematic showing the two phases of the 
in-place rectilinear to cylindrical transformation process, 
according to the present invention. 

FIG. 34 is a block diagram of the Displacement Estima- 15 
tion architecture of FIG. 21 according to the present inven- 
tion. 

FIG. 35 is an illustration of vertical and horizontal down- 
sampling of an image, according to the present invention. 

FIG. 36 is an illustration of the sum-of-absolute differ- 20 
ences (SAD) between a pair of images according to the 
present invention. 

FIG. 37 is an illustration for determining the optimum 
displacement corresponding to the minimum local minima 
in the profile, according to the present invention. 25 

FIG. 38 is a block diagram of the Color Correction 
architecture of FIG. 21 according to the present invention. 

FIG. 39 is a schematic illustrating the color correction 
applied at various regions of the Previous Buffer and Current ^ 
Buffer, according to the present invention. 

FIG. 40 is a block diagram of the Stitching and Blending 
architecture of FIG. 21, according to the present invention. 

FIG. 41 is a schematic illustrating the regions of the 
Previous and Current Buffer, during image compositing, 35 
according to the present invention. 

FIG. 42 is a schematic illustrating the regions of the 
Previous and Current Buffer being composited over a nar- 
rower region, according to the present invention. 

FIG. 43 is a schematic illustrating the regions of the 40 
Previous and Current Buffer contributing to the final 
panorama, according to the present invention. 

FIG. 44 is a schematic illustrating the Current View 
within the Panorama, according to the present invention. 

FIG. 45 is a high level block diagram of an architecture 45 
for a system using a motion playback device according to an 
aspect of the invention. 

FIG. 46 is a diagram illustrating a motion picture simu- 
lation principle, according to the invention. 

FIG. 47 illustrates cylindrical and planar picture repre- 50 
sentations of a panoramic picture, according to the present 
invention. 

FIG. 48 is a detailed block diagram of the motion play- 
back device internal architecture of FIG. 45, according to the 
present invention. 

FIG. 49 shows an X mapping block architecture of FIG. 
49, according to the present invention. 

FIG. 50 shows a Y mapping block architecture of FIG. 49, 
according to the present invention. 60 

FIG. 51 is a block diagram of the picture processor of 
FIG. 49, according to the present invention. 

FIG. 52 shows a vertical subsampling architecture, of the 
picture processor of FIG. 50, according to the present 
invention. 65 

FIG. 53 shows an embodiment of a cubic interpolation 
system of FIG. 51, according to the present invention. 
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FIG. 54 shows an embodiment of a linear interpolation 
system of FIG. 51, according to the present invention. 

DETAILED DESCRIPTION OF AN 
EMBODIMENT 

However, it should be understood that these embodiments 
are only examples of the many advantageous uses of the 
innovative teachings herein. In general, statements made in 
the specification of the present application do not necessarily 
limit any of the various claimed inventions. Moreover, some 
statements may apply to some inventive features but not to 
others. In general, unless otherwise indicated, singular ele- 
ments may be in the plural and visa versa with no loss of 
generality. 

1 . Glossary of Terms Used in this Disclosure 

Actual Next Picture-the picture that is actually captured 
as the next addition to the set of pictures constituting the 
Panorama. 

Bottom-to-Top Vertical Panorama— a Panorama captured 
by taking a set of pictures by rotating the camera up (from 
bottom to top) between each capture, with as little horizontal 
displacement as possible. 

Current Buffer — a cylindrically warped version of the 
current picture. 

Current Picture/Current View/Current Frame-a picture 
displayed on the camera LCD screen and that is updated in 
real time. If the LCD screen is not used or if there is no LCD 
screen, it is the picture that would be captured at any given 
moment if the capture button was pressed. 

Ideal Next Picture-in a Left-to-Right Horizontal 
Panorama, the picture that would be obtained if the camera 
was positioned so that the Previous Picture and Ideal Next 
Picture have an Overlay Part Length equal to the Set Overlay 
Part Length and no vertical displacement. In a Left-to-Right 
Horizontal Panorama, the overlay part is on the right of the 
Previous Picture and on the left of the Ideal Next Picture. 

Ideal Position of the Camera for the Ideal Next Picture 
The position of the camera that allows to capture the Ideal 
Next Picture. 

Image Stitching— the process of digitally combining 
scenes taken from different camera orientations. 

Left-to-Right Horizontal Panorama-a Panorama captured 
by taking a set of pictures by rotating the camera clockwise 
(from left to right) between each capture, with as little 
vertical displacement as possible. 

Overlay Part Length— in a Horizontal Panorama, the width 
of the overlay part. It is expressed in terms of percentage of 
the whole picture width. 

Overlay Part of A Picture-the part of the picture picturing 
the overlay zone. 

Overlay Zone of Two Pictures-the part of a scene that is 
present in the two pictures. 

Panoramic— an image with at least one dimension such as 
height or width which is greater in dimension of a single 
capturing device and often involves a series of images. A 
picture created from a set of pictures and that has at least one 
dimension bigger than the corresponding dimensions of a 
source picture. 

Preview Strip— a computed image created through digital 
processing of the overlay part of the Previous Picture and 
that strives to predict what the overlay part of the Ideal Next 
Picture will look like. 

Previous Buffer— a cylindrically warped version of the 
Previous Picture. 

Previous Picture/Previous Frame-a picture that has 
already been captured and that is the latest addition to the set 
of pictures constituting the Panorama. 
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Right-to-Left Horizontal Panorama-a Panorama captured 
by taking a set of pictures by rotating the camera anticlock- 
wise (from right to left) between each capture, with as little 
vertical displacement as possible. 

Set Overlay Part Length— a constant Overlay Part Length 
for each pair of pictures constituting the Panorama. The Set 
Overlay Part Length is the fixed length chosen for a given 
Panorama. 

Top-to-Boffom Vertical Panorama— a Panorama captured 
by taking a set of pictures by rotating the camera down (from 
top to bottom) between each capture, with as little horizontal 
displacement as possible. 
2. Picture Stitching Camera 

Referring now in more detail to the drawings in which like 
numerals refer to like parts throughout several views, shown 
in FIG. 1 is a block diagram of a digital still camera 100 
according to one aspect of the present invention. A digital 
camera comprises of one or more optical lenses 102 with an 
autofocus/shutter, driver and actuator 104 and associated 
photometry interface 108 such as auto focus, auto shutter and 
contrast control. An imager 106 such as an CCD or equiva- 
lent 104 converts an image projected through optical lens 
102 to a series of pixels 106. Regulators and image drivers 
108 allow regulation of the imager 106. An audio acquisition 
device 212, such as microphone, along with audio process- 
ing circuitry 214, allows a user to make aural recordings 
along with digital images. A battery 118 with power man- 
agement circuitry 118 allows the camera 100 to work as a 
portable device. A picture processor 116, provides pixel 
information to one or more frame buffers 118 coupled to 
picture stitching device 120 which is described further 
below. In this embodiment, the picture stitching device 120 
is implemented as an ASIC. A LCD display 122 or equiva- 
lent enables a user to view the image projected through lens 
102 into imager 106 and controlled by LCD controller 134. 
A PAL/NTS C 124 encoder provides an interface to other 
display types. An image CODEC 132 coupled to picture 
processor 116 provides known image enhancement effects 
for the picture processor 116. A DSP 124 such a STMicro- 
electronics ST-20/SH3-DSP is used to control the memory 
interface and the data 1/0 126 such as Infra-Red, Universal 
Serial Bus or other interfaces. A DRAM provides execution 
memory for the DSP 124 to perform Image Stitching algo- 
rithms as described below. An audio output 130 such as a 
speaker provides the user aurally playback. All of these 
components are representative components of the digital 
camera 100. Storage media 136 such as Flash memory, 
diskettes or removable hard drives store each image and 
associated audio. In one embodiment, the algorithms carry- 
ing out the steps for Image Stitching described below are 
stored on the storage media 136 along with captured images 
and audio. Processing for the images may occur prior to or 
after the image is stored in storage media 136. The general 
operation of a digital camera comprising most elements 
described herein is well understood by those skilled in the 
art. 

One or more user inputs via the LCD Controller 128 
provides user control over camera functions such as the 
orientation of the panoramic e.g., horizontal or vertical, and 
the direction of movement such as a Left- to -Right Horizon- 
tal Panoramic, a Right-to-Left Horizontal Panoramic; a 
Top-to-Bottom Vertical Panoramic; and a Bottom-to-Top 
Vertical Panoramic. Other user input such as the optional 
features and desired effects and to set system parameters 
such as: 

Panorama mode on/off. 

Panorama parameters. 
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-Left- to -Right Horizontal mode. 
-Right-to-Left Horizontal mode. 
-Top-to-Bottom Vertical mode. 
-Bottom-to-Top Vertical mode. 
5 -Set Overlay Part Length. 
Preview display on/off. 
Mixing mode parameters. 
-Alpha blending on/off. 

-Alpha blending parameters such as alpha blending ratio. 
10 -Interlaced block mode on/off. 

-Interlaced block pattern selection. 

In another embodiment, many of the components of FIG. 
1 are embedded in as part of a standard cell in a semi/custom 
semiconductor device. FIG. 3 is an block diagram 300 of the 

15 picture stitching device of FIG. 2 in another embodiment 
integrated as a standard cell In a semi-custom semiconductor 
device, according to the present invention. In this 
embodiment, the picture stitching device 124 and picture 
processor 120 are physically implemented as one unit. The 

20 DSP 134, serial I/O 136, image CODEC 132, LCD preview 
and display 126, PAL/NTSC encoder 130 and LCD control- 
ler & drivers 128 are all integrated as one device 200. To 
those skilled in the art, it will be obvious to substitute and 
modify this exemplary single device 300 to include the 

25 picture stitching device 124 and DSP 224 with other cir- 
cuitry as well. 

3. Panoramic Capturing Overview 

Turning now to FIG. 3 shown is a series of four images 
300 labeled A, B, C, and D with overlap edges parts ab, be, 

30 cd between. These four images are joined together to form 
one Panoramic image. It is important to note that the 
Panoramic images can. be from Left-to -Right as shown in 
FIG. 3 or Right-to-Left for a Horizontal Panoramic. For a 
Vertical Panoramic, (not shown) the series of images can run 

35 from top-to-bottom or from bottom-to -top. Accordingly, this 
present invention is not limited to one type of horizontal or 
vertical orientation, and four types of panoramic images are 
disclosed: a Left-To -Right Horizontal Panoramic; a Right- 
to-Left Horizontal Panoramic; a Top-to-Bottom Vertical 

40 Panoramic; and a Bottom-to-Top Vertical Panoramic. 

FIG. 4 is a flow diagram 400 of the overall image 
acquisition using the digital still camera of FIG. 1, according 
to the present invention. After this overall process is 
described, a more detailed description of the process steps of 

45 FIG. 4 are described below. The digital camera 100 acquires 
a series of images to form a panoramic image, such as the 
series of Left-To-Right Horizontal Panoramic A, B, C, D 
with overlap edge parts ab, be, cd of FIG. 3. The overall 
image acquisition process begins with the initialization of 

50 the digital camera 100. The user inputs from the LCD 
controller setting up the orientation of the panoramic, such 
as horizontal or vertical and other system parameters for 
picture stitching device 124 are set, step 402. The next image 
is located in the view finder of digital camera 100. Returning 

55 to the example in FIG. 3, if the image is A, it is the first 
image, if the image is B, C, or D it is the next image and the 
regions of overlap ab, bd, and cd for each image pair is made 
as a composite. Once the image, either the first frame or the 
successive frames, with the desired overlap are aligned in 

60 the view finder of digital camera 100, the so called Ideal 
Next Picture, is created. The region of overlap, called the 
Preview Strip, is a composite of both the Previous Picture 
and the Current Picture. The Preview Strip comprises a 
perspectively corrected Previous Picture in the region over- 

65 lap. The Preview Strip assists the user in aligning the Current 
Picture and the Previous Picture. The Current View is 
stitched together with the Previous View, in the stitch frame 
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step, 408. A test is made to see if the last image had been 
stitched. Referring again to FIG. 3 that is whether image D 
has been stitched with image C using overlap region cd. If 
the last image has not been stitched then the process returns 
to step 404 until the last image has been stitched, step 410. 5 
Once the last image is stitched final corrections to the overall 
image and overall data is made, step 412. After the first 
frame is acquired, a perspectively-corrected strip of the 
Previous Frame is overlaid on the Current Frame to aid the 
user in acquiring the Current Frame. For example, if a 10 
Horizontal Panorama is being acquired from left- to -right, a 
strip from the right portion of the Previous Frame is per- 
spectively corrected and overlaid on the left portion of the 
Current Frame in the viewfinder. After each frame is 
acquired, the stitch module is invoked with the new frame 15 
buffer. The stitch module aligns and blends the new frame 
into the Panorama, and incrementally saves out the Pan- 
orama. After the last frame is acquired and stitched, the 
Finalize Stitch module is called to save out any remaining 
data. 20 

Turning now to FIGS. 5-8 is an illustration of a series of 
Left- to -Right Horizontal Panorama images illustrating the 
perspective correction of FIG. 4 as seen by the user of the 
digital camera 100, according to the present invention. Note 
the capturing of the Current Picture with an Ideal Next 25 
Picture guided by the overlap generated edge 602. In FIG. 5, 
an image is captured in the digital camera 100, this becomes 
the Previous Picture 500. A Preview Area 602 is shown from 
the Previous Picture 502 to user in FIG. 6. FIG. 7 illustrates 
how the Preview Area 602 is perspective ly corrected or 30 
warped 702 to assist the user in aligning the Current Picture 
800 in FIG. 8. Notice the perspectively corrected Preview 
Area 702 makes aligning the Current Picture 800 much 
easier. As shown in FIG. 8, a Preview Strip 702 is generated 
from the overlay part 602 of the Previous Picture 500. Note 35 
the similarity between the generated Preview Strip 702 and 
the Current View 800 overlay part. The computed Preview 
strip 702 is used as a visual clue, displayed on the LCD 126 
of digital camera 100, to help the user position the camera 
100 until the Current View 800 matches the Preview Strip 40 
702. 

The computed Preview Strip 702 can be displayed on the 
digital camera 100 in various ways to help the user in 
precisely placing the camera in the desired position. One 
way is to alpha blend (with adjustable coefficient) the 45 
preview with the current display of the scene on the LCD 
126. In another embodiment, the interlace some blocks of 
the preview with the displayed Current View on the digital 
camera 100. Further information on blending is described 
below. 50 
4. High Level Picture Stitching Device Architecture 

FIG. 9 is a high level block diagram 900 of a picture 
stitching device 124 of FIG. 1, according to the present 
invention. The Frame Buffers 122 comprise a Previous 
Buffer 902 (or prevBuf) and a Current Buffer 904 (or 55 
curBuf). The Previous Buffer 902 and Current Buffer 904 are 
coupled to the picture stitching device 124 through a set of 
address 906 and data lines 908. A clock input 912 provides 
timing inputs to the picture stitching device 124. A picture 
input from the imager 106, provides the digital image to the 60 
picture stitching device 124. An output labeled out picture 
916, provides an outputfrom the picture stitching device 124 
to other devices such as the Image CODEC 132 and DSP 
134. An input labeled Control settings 910, provides param- 
eter input to picture stitching device 124. Parameter input 65 
includes, the size of the Previous Buffer 902 and the Current 
Buffer 904, the amount of overlap region between images, 
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types of color correction, and more. The Previous Buffer 902 
is a memory devices containing the Previous Picture, its 
sub -sampled version and all the information related to it and 
required by the algorithms. The Current Buffer 904 is 
memory devices containing the Current Picture, its sub- 
sampled version and all the information related to it and 
required by the algorithms. The two buffers can be imple- 
mented either as two separated physical devices or one 
physical device and in this case the distinction is logical 
only. The stitching device requires four types of buffers of 
different sizes: (1) two working buffers each the size of the 
image frame of the digital camera; (2) one buffer for the 
overlay area or Preview Strip; (3) four down sample work- 
ing buffers for motion estimation, described later on; and (4) 
several other smaller buffers and registers for calculations. 
The detailed process flow of stitching using picture stitching 
device 124 is now described. 

A. Stitching Overview The stitching overview of FIG. 4 is 
now described in further detail. FIG. 10 is a detailed flow 
diagram of the stitching overview of FIG. 4, according to the 
present invention. Each of these process steps are described 
further below along with an exemplary hardware implemen- 
tations. To aid with the understanding of the concepts in the 
process flow of FIG. 10 for the Previous Buffer 902 and 
Current Buffer 904 references are made to FIG. 18 which 
illustrates the areas of measurement for two adjacent images 
of FIG. 3, such as images B and C. The stitching process is 
managed around the two working frame buffers 122 the 
Previous Buffer 902 and the Current Buffer 904. In general 
the Previous Buffer 902 contains a cylindrically warped 
version of the Previous Picture and the Current Buffer 904 
contains a cylindrically warped version of the Current 
Picture 800. To illustrate the digital image contents of the 
Previous Buffer 802, the term prevBuffer 1802 is used. The 
prevBuffer 1802 has a preFarEdge 1804 which corresponds 
to the left edge of the Previous Buffer 902 and a preNe- 
arEdge 606 which corresponds to a right edge of the 
Previous Buffer 902. The curBuffer 1808 is a graphical 
illustration of the digital image contents stored in Current 
Buffer 904. The curBuffer 1808 has curFarEdge 1810 which 
corresponds to the left edge of the Current Buffer 904 and 
curNearEdge 1812 which corresponds to the right edge of 
the Current Buffer 904. In general the near and far edges do 
not have to correspond to the true edges of the Previous 
Buffer 902 or the Current Buffer 904. In a vertical panoramic 
orientation, the curFarEdge 1810 and the curNearEdge 1812 
correspond to the top edge of the Current Buffer 904 and the 
bottom edge of the Current Buffer 904 respectively. The 
Previous Buffer 902 also has the top edge and bottom edge 
correspond to the prevFarEdge 1804 and the prevNearEdge 
1806. The overlap region 1818 is defined by the curFarEdge 
1810 on the left and the preNearEdge 1806 on the right. 

The process flow of stitching is divided into two cases: (1) 
stitching the first frame, and (2) stitching the subsequent 
frames. The process flow begins, step 1002 and a determi- 
nation if the first frame has been captured, step 1004. If the 
frame is the first frame, such as A of FIG. 3, the Current 
View is loaded directly into the prevBuffer 1802, step 606. 
In step 1008, the desire amount of overlay between succes- 
sive images is computed, such as the overlay regions ab, be 
and cd of FIG. 3. In one embodiment, the amount of overlay 
is a system parameter that is set by the user. An in place 
transformation such as a rectilinear in place transformation 
to perspectively correct the Overlay Buffer 1818 
(ovrly Buffer). 

The overlay region 1818, also referred to as the Preview 
Strip 702, contains a picture obtained from the previous 
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picture and which has been transformed digitally in order to 
match the perspective for the ideal current picture. This 
overlay picture contained in the overlay region 1818 is used 
to guide the user in position in the camera so that he/she will 
capture a current picture as close as possible to the ideal 5 
current picture. The image contained in the overlay region 
1818 can be obtained from the previous picture in cylindri- 
cal representation or from the previous picture in rectangular 
representation. Obtaining the overlay picture from the cylin- 
drical representation does not require any additional pro- 10 
cessing block, for it is the same transformation used in the 
Motion Picture Playback Module described further in Sec- 
tion 6 "Motion Play-Back of Still Picture That Have Been 
Previously Save", below. However, in the this embodiment, 
the overlay region 1818 is in a rectangular representation. 15 
The use of a rectangular representation yields a better 
picture quality since only one digital transform is used while 
two are needed when using the cylindrical source. The use 
of an in-place cylindrical transform, means that the rectan- 
gular coordinate picture is overwritten by the cylindrical 20 
coordinate picture. And it is necessary to perform the 
computation and saving of the overlay region 1818 in the 
overlay buffer prior to or before the in-place rectilinear- to - 
cylindrical transform occurs. Once the overlay region 1818 
is estimated to correct perspective for the next image is 25 
calculated, step 610. The parameters for stitching are set as 
follows: 

preFarEdge=0; 

preNe arEdge =length; 

prevDispl=de fault. Where length is the length of overlap 30 
between the images, and default is a displacement 
value whose value is set to be greater than zero and less 
the length of overlap. The value is the displacement 
along normal to the direction of orientation for the 
panoramic. Stated differently, for a horizontal 35 
panoramic, the default would be the maximum vertical 
displacement. And the stitching for this first frame is 
complete, step 1016. The Current View is loaded into 
the curbuffer 1808, step 1019, a preview of the cur- 
buffer 1808 into the overlay region 1818 of the Current 40 
Buffer 904 is done to assist the user in aligning the two 
images at the region of overlap. Returning to FIG. 3, 
the region of overlap between the first image A and the 
second image B is ab. The user is trying to capture the 
Ideal Next Picture by positioning the digital camera 45 
100 to align the region of overlap using a perspectively 
corrected Preview Strip 702. As described below, the 
Previous Picture 500 and the Current Picture 800 at the 
region of overlap are superimposed, one on top of the 
other to assist the user in the Ideal Position of the 50 
camera 100. Once the image is captured, it is cylindri- 
cally warped in place, step 1022. At this point both the 
prevBuffer 1802 and the curBuffer 1808 are in cylin- 
drical coordinates. Next, the displacement between the 
previous and current picture (also called motion 55 
estimation), between the prevBuffer 1802 and the cur- 
buffer 1808 is computed along with color correction 
between the prevbuffer 1802 and the curBuffer 1808, 
step 524. In one embodiment, the color correction is 
performed on the prevBuffer 1802 between prev- 60 
FarEdge 1804 and preNe arEdge 1806. And at the same 
time, color correction is also performed on curbuffer 
1808 from curFarEdge 1810 up to the curNe arEdge 
1812. Color correction is not performed on the region 
of prevBuffer 1802 that has already been saved out, 65 
step 1014. After the color correction the overlap region 
of the two buffers are linearly composited, steps 1026 



and 1028. Note in another embodiment, the color 
correction is performed in combination with areas of 
the picture already save out, to yield a smoother color 
variation since the color change could gradually 
speared on a bigger surface. 
Next, the prevBuffer 1802 from prevFarEdge 1804 to 
preNearEdge 1806 with preDisp (previous displacement or 
previous motion estimation) is saved, step 1030. The region 
of prevBuffer 1802 between prevFarEdge 1804 and pre- 
vNearEdge 1806 is save out to image memory, such as 
storage media 142. Since all the prevBuffer 1802 would 
have been saved out a this time, the contents of Previous 
Buffer 1602, that is the image prevbuffer 1802 are no longer 
needed. Now the curbuffer 1808 becomes the prevBuffer 
1802 and the prevBuffer 1802 becomes the curbuffer 1808. 
Stated differently the curBuffer 1808 and prevBuffer 1802 
are swapped. Finally, the system parameters are set as 
follows: 

prevFarEdge=length-curFarEdge; 

prevNearEdge=length; 

prevdisp=curDisp. 

Now further details on the Picture Stitching Device 124 
are described. The description begins with an exemplary 
hardware implementation followed by examples of the pro- 
cesses implemented by each hardware section. 
B. Mixing the Preview Strip With the Current View 

There are several methods to mix the preview strip with 
the Current View. Methods that are discussed are: Alpha 
Blending; Alpha Blending with Progressive Ratio; Interlac- 
ing; Alpha Blending & Interlacing. 

The alpha blending mixing between the preview strip and 
the Current View is illustrated in FIG. 11. The preview strip 
is alpha-blended with the overlap part of the real-time 
Current View and displayed. The alpha-blending operation 
is done with the following algorithm: 

Result=alpha x Preview Strip+(l-alpha) OverlapPart of_Cur- 

rent View. 

The alpha ratio of the blending can be user selected of 
fixed. The alpha blending can be uniform over the preview 
picture or varies over the overlay part as shown in FIG. 11, 
where the Current View is denoted in black 1102 and the 
preview strip denoted in white 1104. The result of the 
constant ration alpha blending in shown in FIG. 12. FIG. 13 
is an illustration of the results of alpha blending of FIG. 11 
with a progressive ratio according to the present invention. 
An error in positioning the camera will be immediately 
noticed on the display as a doubling feature of the viewed 
scene. 

Blocks of the preview picture are interlaced with the 
real-time Current View displayed on the camera. Several 
block sizes can be used, including blocks as wide as the 
preview picture (blocks of lines) and blocks as high as the 
preview picture (blocks of columns). The size of the blocks 
does not need to be uniform. FIG. 14 is an illustration of the 
results of interlace blending of FIG. 11 using blocks and 
FIG. 15 is an illustration of the results of interlace blending 
of FIG. 11 using lines according to the present invention. An 
error in positioning the camera will be immediately noticed 
on the Display as a breaking of features of the viewed scene 
at the blocks edges. 

Both alpha Blending and Interlacing can be used as shown 
FIG. 16, which illustrates the results of alpha blending with 
interlace blending of blocks of FIG. 15 and FIG. 17 which 
illustrates the results of alpha blending with interlace blend- 
ing of only some of the blocks of FIG. 15 according to the 
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present invention. Alpha blending on blocks are illustrated 
with the vertical dimension equal to the vertical dimension 
of the picture. 

C. Saving the Panorama FIG. 19 shows the individual frame 
buffers laid out on a Panorama. The width of the Panorama 5 
grows as more frames are acquired. In one embodiment, the 
height of the Panorama is fixed, and is defined as a function 
of the height of each individual frame and the maximum 
vertical displacement. A fixed vertical height enables the 
calculation of an individual frame, combined with the pre- 10 
vious vertical displacement of the previous frames so that if 
the frame extends beyond the fixed heigh of the panorama is 
clipped. In another embodiment, the height of the panorama 
can be adjusted when the vertical displacement of an indi- 
vidual frame combined with the vertical displacement of all 15 
previous frames is such that the frame extend beyond the 
current height of the panorama. If the vertical displacement 
of an individual frame is such that the frame extends beyond 
the height of the Panorama, the frame is clipped. 

The Panorama is saved in fixed size strips as illustrated in 20 
FIG. 20. The use of saving the image as strips helps facilitate 
the playback of the strip, as described in Section 6 "Motion 
Play-Back of Still Pictures That Have Been Previously 
Stored," below. The saving process is managed by a Pan- 
orama Saving Module that maintains a working buffer the 25 
size of a single strip. Each time a region of prevBuffer 1802 
is saved, as many strips as necessary are saved out, and the 
remainder is stored in the working buffer (to be saved out the 
next time). After the last curBuffer 1808 is saved, the 
(partial) working buffer is saved out. 30 
5. Detailed Picture Stitching Device Architecture FIG. 21 is 
a detailed block diagram 2100 of the picture stitching device 
124 of FIG. 19, according to the present invention. The I/O 
Sections 2102 contains all the Interfaces and data formatting 
resources able to make the device communicate with the 35 
environment. Two lines 2134 and 2136 from Previous Buffer 
1902 and Current Buffer 1904 are coupled to the Alignment 
Preview 2104, Displacement Estimation 2106. Color Cor- 
relation Parameters Estimation, Stitching & Blending 2110 
and the Picture- through-Picture 2120. The output of the 40 
Alignment Preview is fed-back to the I/O Section 2102. The 
output of the Displacement Estimator 2106 and the Color 
Correction Estimator 2108 is fed to a Register 2112, which 
feeds the result into the final Stitching and Blending 2110. 
The video Mode & Geometric Mapping 2116 receives 45 
picture date from I/O Section 2102. The output of the 
Stitching & Blending 2110 and the Picture -through-Picture 
2120 and the Video Mode & Geometric Mappings is fed 
through Selector 2114 which is controlled by Function 
Selection and Control 2118. The function selection and 50 
control 2118 include on/off controls 2138 to each major 
section as shown. The architectures of Alignment Preview 
2104, Displacement Estimation 2106, Color Correlation 
Parameter Estimator 2108, and Stitching and Blending 2110 
are now described followed by the process implemented by 55 
each. 

In an alternate embodiment, everything except the mini- 
mum circuitry of the picture stitching device 124 to enable 
alignment preview 2104 can be located on a remote pro- 
cessing device such as a personal computer or other 60 
microprocessor-based device. The use of a remote process- 
ing device, reduces the size of the electronic circuitry in 
picture stitching device 124, the weight of the picture 
stitching device 124 and the complexity and associated cost 
of manufacturing the picture stitching device 124. In this 65 
embodiment, any or all of the following components may be 
in a remote processing system, including: the displacement 
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estimation 2106, the color correction parameter estimation 
2108, and the stitching & blending 2110 can be located or 
implemented in a remote microprocessor-based system (not 
shown). The interface to the remote microprocessor-based 
system may be through the Serial I/O 136 or Storage Media 
142 that may be removable and coupled to the remote 
microprocessor-based system. 
A.l. Alignment Preview Architecture 

FIG. 22 is a block diagram 2200 of the architecture of the 
Alignment Preview of FIG. 21, according to the present 
invention. After the Previous Picture and Current Picture 
have been converted into cylindrical coordinates, the rela- 
tive displacement between them must be computed. 
A.2. Perspective Correction in Alignment Preview 

An overview of the picture stitching device 124 is now 
described for the region of overlap. FIG. 23 is a top view of 
the Alignment Preview geometry according to the present 
invention. A Previous Picture 2302 was captured by digital 
camera 100 and that f is the focal length of the lens 102 of 
this camera. This Previous Picture 2302 is stored in the 
Previous Buffer 902. The digital camera 100 is rotated by an 
angle 80 so that the Current Picture 2304 is captured with 
a given overlap region along the x axis with the Previous 
Picture 2304. The symbol 8© denotes the rotation angle 
required to produce an overlap of the given width. The 

symbol x ovlp 2310 denotes the right edge of the overlay 

region; x ctr 2308 denotes the center of the image plane. It 

is important to note, that in this embodiment, the rotation is 
horizontal, from left-to-right but the principle applies 
described herein apply to horizontal right-to -left rotation as 
well as top-to-bottom or bottom-to-top rotation. 

The part of each picture (Previous Picture 2302 and 
Current Picture 2304 that is also depicted in the other picture 
is called the overlap region. Although the scene captured in 
the Previous Picture 2302 and the Current Picture 2304 is the 
same in the overlap region 2306 because of a difference in 
perspective. The Alignment Preview 2304, generates a pre- 
view of the Current Picture 2304 by correcting the perspec- 
tive of the overlap region 2306 of the Previous Picture 2302 
to conform with the perspective of the Current Picture 2304. 

The process of perspective correction during the align- 
ment previous involves the following steps: 

(a) Mapping each of the pixels of the Previous Picture 
2302 to an X-Y coordinate; 

(b) Mapping each of the pixels of the Current Picture 2304 
to an X-Y coordinate so that the x-axis is the center of 
the image plane of the current picture along the direc- 
tion of the panning motion of the digital camera 100 
and so that the y-axis is perpendicular to the x-axis and 
lies in the same image plane. This is shown graphically 
in FIG. 23. 

(c) Calculating the rotation angle 8© based on the amount 
of overlap in the preview area along the x direction. 

(d) Calculating the rectilinear x-coordinate transformation 
based on 8© to project or perspectively warp, the 
Current Picture 2304 onto Previous Picture 2302. This 
is shown graphically in FIG. 24. 

(e) Calculating the y-scanline rescaling. Steps c, d, and e 
above are all now shown mathematically with reference 
to FIGS. 24 and 25. 

FIG. 24 is an illustration 1500 of pixels in the overlap 
region of the current picture of FIG. 23 being projected back 
to the Previous Picture of FIG. 23 in the overlap region 2306 

according to the present invention. We note x dst and y dst 

the coordinate of the pixels of the Current Picture 2304 and 

we note x src' and y src' the coordinate of the pixels of 

the Previous Picture 2302. 
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In this example, the overlap region 2306 of the Current 
Picture 2304 indicates the overlay region 2306 that is 
derived from the Previous Frame 2304 in this case, the 
camera 100 rotating clockwise. 

To calculate the rotation angle 6© based on x ovlp 2310: 

6© =arctan(x ctr-x ovlp/f)+arctan(width-x ctr/f). 

The width of this overlap region 2306 is a system param- 
eter. Given the width of the overlap region 2306, the rotation 
angle required to produce an overlap of that width is 
calculated. The width of the overlap region is specified by 
the location of the region's rightmost (innermost) edge 
x ovlp 2310. 

The transform induced on the Previous Picture as a result 
of the 3D rotation is a uniform rescaling of each vertical 
scanline independently. (In the case of camera tilt, the 
transform is a uniform rescaling of each horizontal scanline 
independently). This transform is described by: (a) the 
location of the source scanline corresponding to each des- 
tination scanline; (b) the scale factor for each scanline; and 
(c) column scaling. Calculating the x-coordinate transfor- 
mation: 

x'src=(x dst xcos 6©)+(f xsin &©), 

f src =(-x dst xsin 6©+(f xcos 60). 

Solving for x src: 

x src=( f xx' src)/f src 

EQ1: x_src=f x((x_dst xcos 6©)+(f xsin 8©))/((-xdst xsin&©)+ 
(f xcos 60)) 

The location of the source scanline (x src) is dependent on 

the rotation angle (60) the focal length (f), and the desti- 
nation scanline location (x dst). The equations above 

describe a 2D plane rotation of the x and f axis; the y axis 

is left unchanged. The source scanline location (x src) is 

calculated by projecting onto the image plane of the Previ- 
ous Frame. Calculating y-scanline rescaling: 

y' src=y dst. 

y src=(f xy' src)/f src 

EQ2: y_src=y_dst x[f/((-x_dst xsin 6©)+(f xcos 6©)] 

This equation shows that y src is a uniform rescaling of 

y by a factor that depends on x dst, the rotation angle 6©, 

and the focal length f. Like x src, y src is computed by 

projecting onto the image plane of the Previous Frame. 

Note that there are likely to be instances where the overlap 
region 2306 of Current Picture 2304 cannot be computed 
from overlap region 2306 of the Previous Picture 2302, 
according to the present invention. This is shown as areas 
2502 and 2504 of FIG. 25. 

It should be understood that all trigonometric calculations 
can use look-up tables as required for faster implementation. 
All sections of the picture stitching device 124 use a 'closest' 
coordinate approach when the result of a coordinate com- 
putation is not integers. Another approach is to read the 
value of the neighboring pixels and perform an interpola- 
tion. The location of the source scanline (x src) is calcu- 
lated based on the rotation angle (60), the focal length (f), 

and the destination scanline location (x dst). The principle 

is that, for any given column x dst of the Preview Area 602, 

the location in the Previous Buffer 902 is computed based on 

the corresponding x src column. The given column x dst 

the scaling factor: 
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EQ3: f/((-x_dst xsin6©)+(f xcos 6©)) 

The column scaling is performed by reading a column 
from the Previous Buffer 902 and write a scaled version of 
this column in the Current Buffer 904. 

5 A.3. Fast Buffering During Perspective Correction Turning 
now to FIG. 26, shown is block diagram 2600 of an alternate 
embodiment of FIG. 9 with two optional blocks 2602 and 
2604 used to improve the perspective correction algorithm 
performance, according to the present invention. A buffer 

10 copy section copies a block of memory from the Previous 
Buffer 902 to the Previous Access Buffer 2602. For a given 
Previous Fast Access Buffer size, the size of the Previous 
Buffer 902 part than can be buffered depends on the rotation 
angle (8©), which depends on the desired size of the overlap 

15 area 2306 and of the focal length f . FIGS. 27-30 illustrate an 
example where the complete Previous Buffer 902 and Cur- 
rent Buffer 904 are copied in four passes. The coordinate of 
the area to be buffered in the Current Buffer 904, the 
coordinate of the corresponding points in the Previous 

20 Buffer 902 (given by equations EQ. 1 and EQ. 2 above) and 
the limits of the area to be buffered in the Previous Buffer 
902 are represented. 

FIGS. 27—31 illustrates an example of a limit copy area 
2702 from a Previous Buffer 902 to a Current Buffer 904 of 

25 FIG. 26 during four passes according to the present inven- 
tion. In FIG. 28 which illustrates pass two, note that the areas 
to be buffered in the Previous Buffer 902 overlap between 
two successive passes. Also note that some points of the 
Current Buffer 904 have no corresponding points in the 

30 Previous Buffer 902 as they fall outside the picture area. This 
results in areas of the Current Buffer 904 for which there are 
no information. This is illustrated in FIG. 31 as area 3102. 
B.l. Algorithm for Rectilinear-to- Cylindrical Transform 
(Cylindrical WARP in Place) 

35 FIG. 32 is an illustration of rectilinear and cylindrical 
images. A rectilinear image is parameterized by its optical 
center (x ctr, y ctr) and its focal length (f). The coordi- 
nates of a cylindrical image are the height of the cylinder (h) 
and the angular distance (©). The Rectilinear to Cylindrical 

40 Transform transforms a rectilinear image (an image acquired 
with optics that can be approximated by a model pin-hole 
camera) Into cylindrical coordinates. The specific algorithm 
outlined below performs the transformation in-place with 
the use of a small buffer. It also automatically crops the 

45 result as shown in the right half of FIG. 32. Like the Overlay 
Generator, the Rectilinear to Cylindrical Transform uni- 
formly rescales each vertical scanline in the image indepen- 
dently. Thus, the transform is described by: (a) the location 
of the source scanline corresponding to each destination 

50 scanline, and (b)the scale factor for each scanline. 

FIG. 33 is an illustration of the two phases of the in-place 
rectilinear to cylindrical transformation process. The left- 
to-right phase ends with the last column being the column 
just before the first column corresponding to A col <0; the 

55 right-to -left phase ends with that column. The in-place 
implementation of the Rectilinear to Cylindrical Transfor- 
mation comprises two passes (which could be implemented 
in parallel). The first pass is the left- to-right pass which 
begins with the leftmost column and ends just before the first 

60 column corresponding to A col <0. Before proceeding with 
either passes, the vertical scanline corresponding to the last 
column of the left- to-right pass is buffered. 

During the left-to-right pass, each new destination vertical 
scanline is computed by rescaling a scanline somewhere to 

65 the right of it (or even at the same location). Since the image 
representation of the current implementation could be sub- 
sampled horizontally, a buffer of two column of pixels, the 
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current scanline and the next (right), is needed. When using 
a YcrCb format for the pictures were Chroma samples (Cr: 
Chroma red and Cb: Chroma blue) are decimated. This is 
because the original subsampled chroma samples could be 
required after they have been modified (rescaled). Each 5 
Chroma sample is shared between two successive Luma (Y) 
samples. At the end of each iteration of the pass, the buffers 
are updated. If the source vertical scanline is either the 
current or the next scanline, the buffered copies are used 
instead of being derived from the Current Picture. 10 

In another embodiment, interpolation is used to create a 
column of source pixel (in the case when the computed 
source column coordinate does not correspond exactly to a 
real source column coordinate, the closest real source col- 
umn is used. The use of the closest real source column is an 15 
approximate solution that yields average picture quality. To 
perform interpolation, several source columns are used so a 
few columns are buffered for later interpolation. 

The right-to-left pass proceeds analogously, the only 
difference is that the source scanlines are computed by 20 
rescaling scanlines somewhere to the left of current scan- 
lines. A similar two -scanline buffer that contains the current 
and the next (left) is also required. The last (leftmost) 
scanline could require the scanline to the left of it. If so, the 
copy buffered at the start should be used in place of the 25 
Current Picture. 

B.2. Calculating the X-coordinate Transformation 

The location of each source vertical scanline (x src) 

depends on the destination scanline (©) and the focal length 

(f). The coordinate 0 varies linearly from 0 min to 30 

0 max as shown in the following equations. 

Where 

x src=the coordinate of a column in the source picture 

0=the Angle coordinate of a column in the transformed 

picture 35 

The transformation relation: 

x src=f xtan (©) 

where f is the focal length used to capture the source picture. 4Q 

x src varies from -w/2 to w/2, and is equal to 0 at the 

center of the picture. 0 varies from 0 min to 0 max 

and is equal to 0 at the center of the picture. Note that 

0 max =f xtan (w/2)=-0 min 

For storage purpose, a column equivalent is defined to the 45 
angle coordinate and define x dst is defined such that: 



x dst=©xf xtan (0 max)/© max 

This definition of x dst ensure the following desirable 50 

property: 



for 0=0, x_dst=0; 

for ©=© max, x dst=f xtan(© max)=w/2 

for ©=© min, x dst=f xtan(© min)=-w/2 
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i.e. x dst is centered on the center of the destination scan 

line and varies from -w/2 to w/2, just like x src. 

Using this definition it follows that: 60 

x src =f xtan (©)=f xtan (( x dst/f ) x(© max/tan(© max))) 

to simplify, this can be written 



x src=f xtan (©) x dst), where f and C are constants 

By studying the function x src(x dst), it is follows that: 
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for any x dst between 0 and w/2, x dst >=x src(x dst)>=0 

for any x dst between -w/2 and 0, 0 >=x src(x dst)>=x dst 

Returning to the in place transformation, in order to avoid 
'gaps' or overwriting in the destination image, for each 

column (i.e. for each x dst) of the destination scanline, we 

find the closest corresponding column (i.e. x src) in the 

source image. 

It has been show that: 

for any x dst between 0 and w/2, 

x dst>=x src(x dst) >=0 

This means that, for every x dst, the information 

required is located in the source image at column coordinate 

x src, which is smaller or equal to x dst, never bigger than 

x dst. In turn, this means that, for any given x dst, the area 

of the SOURCE image situated for column coordinate 
bigger than x dst is not needed. 

By scanning the destination image between 0 and w/2 in 
decreasing order (i.e. starting at w/2 and finishing at 0). 

at time tl, compute x dst using x srcl<=x dstl 

at time t2 >tl, compute x dst2 using x src2 <=x dst2 

and x dst2 <x dstl since the scan is progressing in 

decreasing order. 

Therefore, it follows: 

x src2 <x dstl 

All the information of the source image situated at column 

coordinate bigger than x dst, is deleted, since it will not be 

needed for the creation of the current x dst column of the 

destination image nor for the creation of any column later in 
the processing. 

Given proper care, the discarded area of the source image 
can be used to store the destination image itself in order 
to save the cost of a separate buffer. 
When doing the overwriting of the source image by the 
destination image, some special care must be taken, in 
particular: 

since the function x src(x dst) will most of the time 

not return an integer value, other methods may be 
used to find the closest source column matching 

x src. For example, interpolating a source column 

based on adjacent source columns, a small number of 

source columns around x dst should be buffered and 

not overwritten as long as they can be necessary in 
the computation, 
when using a Y,Cr,Cb format (in which Cr and Cb 
samples are decimated, i.e.: shared between several 
Y sample columns) for the source image, proper 

buffering of Y columns must be done around x dst 

in order not to loose relevant information. 
In the region between -w/2 and 0, the same reasoning 
apply if, between -w/2 and 0, the destination image is 
scanned in increasing order (i.e. starting at -w/2 and finish- 
ing at 0). So, to perform the in place cylindrical 
transformation, it is necessary to split the picture processing 
in two part, one from -w/2 to 0 and the other one from w/2 
to 0. Again, some special care (buffering of a few columns 

around x dst) must be taken around x dst=0. 

Although only a small number of columns must be 
buffered and the size of these buffers is small compared to 
the size of a complete image buffer. This means that using 
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in place transform save an appreciable amount of memory, 
and the benefits are proportional to the size of the images 
used. 

B. 3. Calculating Y-scanline Reseating 

5 

y src=y scale x y dst 

y' scale=f / sqrt ( tan 2 ©+1) 

y' scale min=min ( f / sqrt ( tan 2 © min+1), f/ sqrt (tan 2 

©_max+l)) 10 

y scale =y' scanline/ y' scale min 

This equation shows that the destination vertical scanline 
is a uniform rescaling of the source scanline whose scale 
factor depends on the focal length (f) and the horizontal 15 
coordinate (0). The scale factor is normalized such that the 

scale factor corresponds to unity at either the left (© min) 

or the right edge (© max). 

C. l. Algorithm for Displacement Estimation (Motion 
Estimation) 20 

Motion estimation is performed on a pair of images in 
cylindrical coordinates. Motion estimation is carried out in 
four steps: 

(1) Horizontal and Vertical Downsampling; 

(2) One-Dimensional High Pass Filtering; 

(3) Computation of Displacement of Dominant Direction; 
and 

(4) Computation of Displacement in Other Direction. 
During the first step, each image is horizontally and 30 

vertically downsampled. 
Since two images are being downsampled, a total of four 
downsampled images are generated. Each of the four down- 
sampled images is then high pass filtered in the horizontal 
direction. After high pass filtering, the displacement in the 35 
dominant direction is estimated using the appropriate pair of 
downsampled images. For example, if the horizontal dis- 
placements are thought to be larger than the vertical dis- 
placements within their respective ranges, then the horizon- 
tal displacement is estimated first using the vertically 40 
downsampled images. Note it is impossible to be sure if the 
horizontal displacement are larger since the displacements 
have not been calculated yet, however in a horizontal 
panorama, the horizontal displacement are naturally larger 
than the vertical displacement because it is easy to keep the 45 
camera more or less level while it is more difficult to make 
a mistake on the horizontal positioning of the camera for 
each shot. At this point, in one embodiment, the assumption 
is that the vertical displacement is zero. This is possible 
because (a) the vertical displacement is assumed to be small, 50 
and (b) the horizontal displacement estimation is carried out 
on vertically downsampled images. After the horizontal 
displacement is obtained, the vertical displacement is esti- 
mated using the horizontally downsampled images. In this 
case, we take into consideration the previously estimated 55 
horizontal displacement by translating the horizontally 
downsampled images appropriately, 
a. Horizontal and Vertical Downsampling 
FIG. 34 is a block diagram of the Displacement Estimation 
architecture of FIG. 21, according to the present invention. 60 
The horizontal downsampled result is also automatically 
mirror reflected so that a vertical displacement of the origi- 
nal image is equivalent to a horizontal displacement of the 
mirror reflected, horizontally downsampled image. During 
Horizontal and Vertical Downsampling, multiple vertical 65 
scanlines and multiple horizontal scanlines respectively, are 
(block) averaged. During Horizontal Downsampling, the 
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low-pass result is also reflected about y=x; that is, each 
downsampled vertical scanline is stored as a horizontal 
scanline. This is done so that the same one -dimensional 
High Pass Filtering and one-dimensional Displacement 
Computation algorithm could be used. Both images are 
downsampled in the same way, this step could be done in 
parallel. The output of FIG. 34 is the displacement along the 
horizontal and vertical direction of the two pictures. This 
results are stored into the Register 512. The Register 512 is 
a memory element. 

b. One -Dimensional High-Pass Filtering 
One-Dimensional High Pass Filtering is performed next 

on each of the downsampled images (a total of four two per 
image). This high pass filtering is needed to make the error 
measure robust (invariant) to global illumination changes. 
One- dimensional high pass filtering is carried out in the 
horizontal direction only. Since the horizontally down- 
sampled result has been mirror reflected, one-dimensional 
high pass filtering in the horizontal direction is the same as 
filtering in the vertical direction of the unreflected result. 
The current choice of filter coefficients is [-1, 2, -1]. 

c. Estimating the Horizontal and Vertical Displacements 
The motion estimation required for the digital camera 100 

is larger in one direction than the other. When the Panorama 
is horizontal, the horizontal displacement is assumed to be 
larger than the displacement in the vertical direction. 
Conversely, when the Panorama is vertical, the vertical 
displacement is assumed to be larger than the displacement 
in the horizontal direction. Thus, and because of the 
downsampling, we can first compute the displacement in the 
dominant direction, assuming that the displacement in the 
other direction is zero. After this displacement is 
determined, the displacement in the minor direction is 
computed taking into consideration the displacement in the 
dominant direction. Nevertheless, displacements are always 
calculated in the horizontal direction because of the prior 
rotation of the horizontally downsampled scanlines. C.2. 
Calculating the SAD Between Images 

FIG. 36 is an illustration of the computation of the 
sum-of-absolute differences (SAD) between a pair of 
images. The global SAD is computed by accumulating the 
local SAD between corresponding pairs of scanlines, over 
all scanlines. In order to determine the optimum horizontal 
displacement between two images, the sum-of-absolute dif- 
ferences (SAD) between the pair of images is computed at 
every possible displacement within the search range. Instead 
of computing the global SAD between two images 
repeatedly, the local SAD (the SAD between corresponding 
pairs of scanlines) is computed for each pair of scanline and 
accumulated in the SAD Accumulator as illustrated in FIG. 
36. This allows the images to be accessed once. The pseudo- 
code for the operation is shown below. 

For each displacement dx: 

SAD[dx]=0. 

For each scanline y: 

For each displacement dx: 

S AD[dx] + = ScanlineS AD(dx, Im age 1 Scanline, 
Image2Scanline). 

Note that the ScanlineSAD function computes the SAD 
between pairs of scanline only over the overlap region. Since 
the overlap region is of varying size, the scanline SAD is 
also normalized; specifically, the SAD/pixel is computed. 
Alternatively, if the window over which the SAD is com- 
puted by ScanlineSAD is held fixed, then normalization is 
unnecessary. Note that the ScanlineSAD for all the displace- 
ments within the search range could be computed in parallel. 
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Although SAD is shown, it is important to note that other 
methods exist to estimate the difference between the two 
pairs of images. For example, correlation is another method, 
although more computationally intensive, that can be used in 
place of SAD. Still other methods exist to accomplish the 5 
same results as SAD know to those of average skill in the art. 

C. 3. Determining the Optimal Displacement 

FIG. 37 is an illustration of the SAD (error) profile. The 
optimum displacement is the displacement corresponding to 
the minimum local minima in the profile. After the SAD 10 
between images has been computed, the SAD accumulator 
is searched for the minimum local minima. The displace- 
ment corresponding to this local minima is the optimum 
displacement. Note that the minimum local minima is used 
rather than the local minima. In effect, this excludes the 15 
minima that could be found on the boundary of the search 
range as these would not correspond to the artifacts pro- 
duced of the local search method. 

The required operations are implemented by the Displace- 
ment Estimation block depicted in FIG. 34. 20 

If the pictures have to be stitched along the horizontal 
direction, then the displacement along this direction (the 
dominant one) is computed. The vertical estimation will be 
computed as a second step. On the other hand, if the 
dominant direction is the vertical one, then the displacement 25 
along this direction will be computed first. The assumption 
in this embodiment is that the dominant direction is the 
horizontal one, however it is important to point out again 
that all the teachings in the current invention apply to 
vertical panorama image capturing as well. The downsample 30 
blocks compute the two sub -sampled pictures along the 
vertical direction and store the result in the respective 
buffers. Then these pictures are pre- processed (for ex. an 
high pass filter is applied to them) and the Sum of the 
absolute differences is computed. Further details on this 35 
algorithm are described in the section below entitled Algo- 
rithm for Image Compositing. 

D. l. Color Correction Architecture 

FIG. 38 is a block diagram 3800 of the Color Correction 
architecture of FIG. 21 according to the present invention. 40 
The Color Correction Parameters Estimation architecture is 
described in FIG. 39. The outputs of these blocks are simple 
sets of values and these values will be used in the color 
Correction block. The values extracted form the current and 
previous pictures are called: Brightness Current (Be), Con- 45 
trast Current (cc), BrightnessPrevious(Bp), ContrastPrevious 
(Cp). These abbreviations are used in FIGS. 38 and 40. 
These values can be stored in any general purpose register 
(not shown). 

D.2. Algorithm for Color Correction Estimation 50 

The color correction model that is adopted in the Camera 
100 involves brightness and contrast histogram matching. 
Other more sophisticated techniques exist but histogram 
matching was chosen because of its independence on motion 
estimation results and could therefore be computed in par- 55 
allel. 

Denote I prev as a color channel of the previous buffer 

(prevBuffer 1802), and I_cur as the same color channel of 
the Current Buffer (curBuffer 1808). Assume a default 
overlap between the previous and Current Buffers; this 60 
default could be the default position of the Current Buffer 
with respect to the previous buffer. Within the area of 
overlaps in the two images, compute: 



E prev=l/N Sum (I prev); E2 prev=l/N Sum (I =13 prev x 

I prev) 

Var prev=E2 prev-(E prev xE prev) 
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E_cur=l/N Sum (I_cur); E2_cur=l/N Sum (I_cur xl_cur) 
Var cur=E2 cur— (E cur xE cur) 

The brightness and contrast to apply to the Current Buffer to 
match the previous buffer is given as: 

Brightness =E prev— Contrast x E cur 

Contrast =sqrt ( Var prev / Var cur) 

Instead of color correcting the Current Buffer only, we apply 
color correction to both buffers equally using the following 
equations: 

Al=(Brightness +Contrast x I)— I 

r=i +pxAi 

Color correction is later applied according to the following 
equations: 

F prev=B rightness pr ev+Contr as t pr evx I p rev 

I' cur =Brightness cur+Contrast curxl cur 

This color correction technique is applied to each color 
channel independently. For example, if the buffers are YCC, 
separate brightness and contrast parameters for each of the 
channel is calculated. 

D. 3. Algorithm for Color Correction 

In this section, a description of the color correction that is 
applied to the previous and Current Buffers. The parameters 
of the color correction are the respective brightness and 
contrast adjustments for each color channel that was com- 
puted earlier by matching the buffers' histograms. 

Turning now to FIG. 40, shown is an illustration of the 
amount of color correction applied at various regions of the 
previous and Current Buffer. 

Color correction is applied to varying degrees in different 
parts of each buffer. Within the region of overlap, color 
correction Is applied fully ((3=1); outside of the region of 
overlap, color correction is applied less further away from 
the overlap (0 <=p>l). 

Brightness prev=Brightness / sqrt(Contrast) 

Contrast prev=l / sqrt (Contrast) 

Brightness cur=B rightness / sqrt(Contrast) 

Contrast cur=sqrt(Contrast). 

A variant of this scheme does not apply full color cor- 
rection even within the overlapping region. Instead, only up 
to 75% of color correction is ever applied. Thus, beta ranges 
from 0 to 0.75 outside of the overlapping region, and is 0.75 
within. Using this technique, the previous and Current 
Buffers are not sufficiently color corrected to match each 
other. This is acceptable because the subsequent image 
compositing step transitions smoothly the previous buffer 
into the Current Buffer. Not completely color correcting the 
two buffers is useful as it produces a smoother color tran- 
sition between the two buffers. 

E. l. Image Splicing Architecture 

FIG. 40 is a block diagram of the Stitching and Blending 
architecture of FIG. 21, according to the present invention. 
E.2. Algorithm for Image Compositing 

FIG. 41 is an illustration showing the regions of the 
previous and Current Buffer during image composition, 
according to the present invention. This section describes the 



US 6,111. 

23 

image compositing between previous and Current Buffer 
that is carried out to construct the seamless Panorama. Since 
the region of the previous buffer between prevFarEdge and 
prevNearEdge will be written out next, the composited 
result is updated onto the previous buffer. Only the overlap 5 
region between the previous and Current Buffers are com- 
posited. The two buffers are composited according to the 
following equation: 

I' prev=al prev+(l-a) I cur ^ 

where a. is a weighting factor that varies linearly between 
one and zero as shown in FIG. 41. 

E. 3. Algorithm for Image Splicing 

This section describes a modification of the image com- 
positing technique described earlier. During image 15 
compositing, the previous and Current Buffers are alpha- 
blended within their region of overlap. While this results in 
a smooth transition between the two buffers, it produces 
ghosting when the content in the two buffers differ. For 
example, if a person is in the overlap region within one 20 
buffer and not the other, a ghost-like reproduction of that 
person will be present in the final Panorama. If the com- 
positing takes place over a narrower region of the overlap 
region to the left or to the right of the person, then the 
resulting Panorama will either contain the person entirely or 25 
not include the person (respectively). FIG. 42 shows an 
illustration of two buffers being composited over a narrow 
region. 

Turning now to FIG. 42, shown is illustration of the 
previous and Current Buffer being composited over a nar- 30 
rower region. The location of the center of the narrower 
compositing region could be taken as the location that yields 
the least SAD error of a given error function. One such error 
function would be the sum of absolute differences across all 
corresponding pixels of each vertical column within the 35 
overlapping region. This would be computed with the Cur- 
rent Buffer at its optimum displacement with respect to the 
previous buffer. The width of the compositing region could 
be fixed or determined from the same error function. For 
example, if the minima of the error function (with which we 40 
determine the location) is narrow, then the width should be 
narrow. Conversely, if the minima is wide, then the width of 
the compositing region should be larger. 

FIG. 43 illustrates the regions of the previous and Current 
Buffer contributing to the final Panorama. Note that when 45 
compositing over the narrower compositing region, the 
prevNearEdge is no longer the right edge of the previous 
buffer. Instead, it is set to be the right edge of the compos- 
iting region. Thus, only the region of the previous buffer 
between the prevFarEdge and the prevNearEdge is saved 50 
out. 

F. f . Algorithm for Cylindrical to Rectilinear Transform 
Turning now to FIG. 44, shown is an illustration of the 

Current View within the Panorama. The Cylindrical to 
Rectilinear Transform transforms a portion of a cylindrical 55 
Panorama into rectilinear coordinates. This transform is 
basically the inverse of the Rectilinear to Cylindrical Trans- 
form described in a previous section. The panoramic viewer 
renders each new view of the Panorama using this transform. 
Each new view is parameterized by the origin in cylindrical 60 
coordinates (© origin), the focal length (f), and the dimen- 
sions of the viewport. Panning the panoramic viewer 

involves changing the © origin parameter and re -rendering 

the new view. 

Like the Rectilinear to Cylindrical Transform, the Cylin- 65 
drical to Rectilinear Transform uniformly rescales each 
vertical scanline in the image independently. Thus, the 
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transform is described by: (a) the location of the source 
scanline corresponding to each destination scanline, and (b) 
the scale factor for each scanline. 
F.2. Calculating the X-coordinate Transformation 

© src=atan2(x dst, f)+© origin 

The location of each source vertical scanline (q src) 

depends on the destination scanline (x dst), the focal length 

(f), and the origin (© origin). 

F.3. Calculating Y- scanline Rescaling 

y src=y scale * y dst 

Each vertical scanline is a uniformly rescaled copy of the 
original. The scale factor is unnormalized by the factor 
y' scale min that was applied in the forward transform. 

y' scale=f/sqrt (tan 2 © src+1) 

=f 2 /sqrt(x_dst 2 +l) 

y' scale min=min (f/sqrt (tan 2 © min+1), f/sqrt (tan 2 © max 

+1)) 

y scale=y' scale min / y' scale 

Thus, the scale factor depends on the location of the desti- 
nation scanline (and ultimately, the location of the source 

scanline), and the focal length. The denominator of y' scale 

can be precomputed as a look-up table, indexed by the 

location of the destination scanline (x dst). 

6. Motion Play-Back of Still Pictures That Have Been 
Previously Stored 

The digital camera has previously stored images such as 
images A, B, C, D of FIG. 3 that have been stitched together 
as described above. The following section describes how 
these previously stored images can be displayed with a 
perspective and motion of a video camera that could have 
captured the images comprising the panoramic scene. It is 
assumed that these images are available. This process of 
motion play-back (MPB) is based upon the continuously 
scanning of the picture in order to simulate the motion of a 
video camera acquiring the scene that is displayed. 

Turning now to FIG. 45, shown is a high level architecture 
of a system 4500 for implementing motion play-back 
(MPB), according to the present invention. An Image Gen- 
erator 4502 contains all the resources (H/W and S/W) to: (f ) 
create the picture by stitching several pictures together and 
then transfer it to the Memory 4504; or (2) up-load a picture 
from a non volatile memory, such as storage media 142, 
decode it and send it to the memory 4504. The memory 4504 
is a standard block that contains the pictures. The memory 
4504 is connected to a motion play-back (MPB) 4506 by 
address 4510 and data lines 4512. The MPB 4506 generates 
the addresses to properly read the picture in order to simulate 
the motion according to the parameters provided with the 
settings 451A. The settings 4514 provides information 
related to: 

* Speed of motion. 

* Frame Frequency. 

* Raster Structure: Interlaced/Progressive. 

* ViewPort Horizontal and Vertical (Xv-Yv) dimension. 

Shown is a viewport (the rectangle with © original) 

where Xv and Yv are the viewport dimensions. 

* Picture Ho rizontal and Vertical (Xv-Yv) dimension. 

* Source Picture Reference System: Cylindrical, Planar, 
Spherical. 
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* Picture Starting address. 

* Focal Length (f) of the optic used to take the picture. 
The picture information is provided to the MPB 4506 by 

the data line 4512 and the MPB 4506 processes them 
according to the settings 4514. 5 

Now an example is provided in order to describe how the 
MPB 4514 works. FIG. 46 depicts a picture having the 
horizontal extent, i.e. along the X direction, equals 2 times 
the horizontal extent of the viewport. It is assumed the 
viewport is an LCD 126 or output, such as PAL or NTSC, 10 
to a TV set with interlaced scanning. Interlace is illustrated 
as even rows 4062 and odd rows 4064. During the first field 
(the odd one, for example) the portion of the picture to be 
displayed is depicted in the left-most part of FIG. 46 in 
window 4606. For the even field (the next one) the portion 15 
to be displayed is shifted by an offset 4608 to show the 
following window 4610. This process is repeated so that the 
viewport (i.e. window 466) is repeatedly moved by offset 
4608 to form subsequent view 5 as desired. The actual 
interfaced even rows 4604 that are displayed for viewing 20 
window 466. Similarly, for viewing window 4610, the odd 
rows 4604 are displayed. 

The MPB is described now based on the following 
assumptions: (1) Raster Structure: Interlaced; and (2) Source 
Picture Reference System: Cylindrical. FIG. 47 is an illus- 25 
tration of the coordinate systems 4700 used for the stored 
pictures 4702 and the raster display pictures 4704, according 
to the present invention. The stored images 4702 are the 
previously stitched frames. In order to properly display a 
picture, the cylindrical-to-planar conversion must be imple- 30 
mented. Generally, this graphic transformation is applied in 
inverse mode, i.e. starting from the position of the destina- 
tion picture (corresponding to the position of a point of the 
display, in this case) the corresponding point in the source 
picture is found. The equations describing the cylindrical- 35 
to-planar mapping applied in the inverse mode are: 

(1) Xs =arctan (X / f) +origin 

(2) Ys =Y_scale xYd 4Q 

(3) Y_scale =sqr(f)/sqrt (sqr(Xd) +1) 

Depending on the speed of the motion to be simulated, an 
offset 468 must be added to the address related to the pixel 
position along the horizontal direction of source picture. 45 
This offset 4608 depends on the Raster structure. For 
example, if the raster is interlaced this offset 4608 must be 
updated for every field. On the other hand, if the raster is 
progressive, this Offset must be updated every frame. Where 
f represents the focal length. Equations (1), (2) and (3) above 50 
can be implementing the mathematical functions with a look 
up table, but this is not restrictive for the invention. FIG. 48 
is a block diagram 4800 of the major components of MPB 
4506 of FIG. 45, according to the present invention. A 
counter 4802 generates the coordinate of a point in the 55 
display coordinates 4704. The FP, 4818 signal directs when 
the offset 4608 must be updated. FIG. 49 is an illustration of 
the X-mapping 4808 for carrying out equation (1). FIG. 50 
is an illustration of the Y-mapping 4806 for carrying out 
equations (2) and (3). Address Mapping 4810 translates the 60 
cylindrical coordinates Xs and Ys of 4702 to planar coor- 
dinates Xd and Yd 4704. 

Other embodiments are possible. For example, if the input 
picture is in planar coordinates, equations (1), (2) and (3) do 
not implement any transformation in the MPB 4506. On the 65 
other hand if the input picture in spherical coordinates the 
MPB 4506 implements a different set of transforms. The 



608 Bl 

26 

physical address of a picture coordinate is used to retrieve a 
pixel from the memory. Due to the nature of the transform, 
it is possible the location Xs and Ys does not correspond to 
a pixel already existing on the source image. This location 
can be between two pixels. To recover this drawback, the 
Pixel Processor block implements an interpolation algorithm 
in order to compute the pixel value at the position Xs, Ys. 
Moreover the Picture Processing 4812 implements the 
down-sampling along the vertical direction in order to fit the 
vertical size of the viewport. 

One architecture of the Picture Processing 4812 is 
depicted in FIG. 51. And one architecture of the Vertical 
Sub-Sampling 5102 is depicted in FIG. 52. The vertical 
Sub -Sampling 5102 requires some Line Memories (LM) and 
Pixel Delays (PD) in order to implement the processing 
window required by the particular Interpolation algorithm 
implemented by the SS Interp. block. Some of those algo- 
rithms require the Field/Frame Signal too. The Interpolation 
block, computes the value of the pixel at the Xs Ys position 
starting from the available ones in the source picture. A 
possible implementation of the Interpolation 5104 using a 
Cubic Interpolator is described in FIG. 53 while FIG. 54 
illustrates a possible implementation of the Interpolation 
block using the linear interpolation algorithm. 

Although a specific embodiment of the invention has been 
disclosed, it will be understood by those having skill in the 
art that changes can be made to this specific embodiment 
without departing from the spirit and scope of the invention. 
The scope of the invention is not to be restricted, therefore, 
to the specific embodiment, and it is intended that the 
appended claims cover any and all such applications, 
modifications, and embodiments within the scope of the 
present invention. 

What is claimed is: 

1. A device for estimating the motion between a first 
image and a second image in a system for creating a 
panoramic image from a plurality of images taken by a 
camera, the device comprising: 

a downsampler for downsampling the: 

first image, in at least a first and a second direction, so as 

to produce a first downsampled image along a first 

direction and a second downsampled image along a 

second direction; and 
second image in at least the first and the second direction, 

so as to produce a second downsampled image along 

the first direction and a second downsampled image 

along the second direction; 
a filter for filtering out any global illumination changes 

between the first image and the second image; and 
a estimator for calculating: 

a first displacement along the first direction between the 
first downsampled image along the first direction and 
the second downsampled image along the first direc- 
tion; and 

a second displacement along the second direction between 
the first downsampled image along the second direction 
and the second downsampled image along the second 
direction. 

2. The device as defined in claim 1, further comprising: 
a converter for converting a pair of images from rectan- 
gular coordinates to cylindrical coordinates to create a 
first image and a second image. 

3. The device as defined in claim 1, wherein the estimator 
for calculating a first displacement is calculated based on the 
displacement along the second direction being zero. 

4. The device as defined in claim 1, wherein the estimator 
for calculating includes a an accumulator for holding the 
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sum-of -absolute-differences (SAD) between a predefined 
area of the first image and a predefined area of the second 
image. 

5. The device as defined in claim 3, wherein the pre- 
defined area is a scan line for the first image and a scan line 
for the second image and the accumulator holds the SAD 
between the scan line the first image and the scan line for the 
second image. 

6. The device as defined in claim 1, wherein the filter is 
a high pass filter. 

7. A method for estimating the motion between a first 
image and a second image in a system for creating a 
panoramic image from a plurality of images taken by a 
camera, the method comprising the steps of: 

downsampling a first image in a first direction and in a 
second direction; 

downsampling a second image in a first direction and in 
a second direction; 

filtering the first and the second image so as to filter out 
any global illumination changes between the first image 
and the second image; 

calculating a first displacement along the first direction 
between the first downsampled image along the first 
direction and the second downsampled image along the 
first direction; and 

calculating a second displacement along the second direc- 
tion between the first downsampled image along the 
second direction and the second downsampled image 
along the second direction. 

8. The method according to claim 7, further the compris- 
ing the step of: 

converting a pair of images from rectangular coordinates 
to cylindrical coordinates to create a first image and a 
second image. 

9. The method according to claim 7, wherein the step of 
calculating a first displacement includes calculating a first 
displacement based on the displacement along the second 
direction being zero. 

10. The method according to claim 7, wherein the step of 
calculating a first displacement includes the sub -step of: 

accumulating the sum-of -absolute-differences (SAD) 
between a predefined area of the first image and a 
predefined area of the second image. 

11. The method according to claim 10, wherein the 
sub -step of accumulating includes accumulating the SAD of 
a scan line for the first image and a scan line for the second 
image and further comprising the step of: 

storing the SAD between the scan line the first image and 
the scan line for the second image. 
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12. The method according to claim 8, wherein the step of 
filter includes the step of high pass filtering. 

13. A computer readable medium containing program- 
ming instructions for estimating the motion between a first 

5 image and a second image in a system for creating a 
panoramic image from a plurality of images taken by a 
camera, the programming instructions comprising: 

downsampling a first image in a first direction and in a 
second direction; 
10 downsampling a second image in a first direction and in 
a second direction; 
filtering the first and the second image so as to filter out 
any global illumination changes between the first image 
and the second image; 
15 calculating a first displacement along the first direction 
between the first downsampled image along the first 
direction and the second downsampled image along the 
first direction; and 
calculating a second displacement along the second direc- 
tion between the first downsampled image along the 
second direction and the second downsampled image 
along the second direction. 

14. The computer readable medium according to claim 
13, further the comprising the programming instructions of: 

25 converting a pair of images from rectangular coordinates 
to cylindrical coordinates to create a first image and a 
second image. 

15. The computer readable medium according to claim 
13, wherein the programming instruction of calculating a 

30 first displacement includes calculating a first displacement 
based on the displacement along the second direction being 
zero. 

16. The computer readable medium according to claim 

13, wherein the programming instruction of calculating a 
3:> first displacement includes the programming instruction of: 

accumulating the sum-of-absolute-differences (SAD) 
between a predefined area of the first image and a 
predefined area of the second image. 

17. The computer readable medium according to claim 
16, wherein the programming instruction of accumulating 
includes accumulating the SAD of a scan line for the first 
image and a scan line for the second image and further 
comprising the programming instruction of: 

45 storing the SAD between the scan line the first image and 
the scan line for the second image. 

18. The computer readable medium according to claim 

14, wherein the programming instruction of filter includes 
the step of high pass filtering. 



